Design Logging Framework

Ashish

Ashish Pratap Singh

medium

In this chapter, we will explore the low-level design of a logging framework in detail.

Lets start by clarifying the requirements:

1. Clarifying Requirements

Before starting the design, it's important to ask thoughtful questions to uncover hidden assumptions and clearly define the scope of the system.

Here is an example of how a discussion between the candidate and the interviewer might unfold:

After gathering the details, we can summarize the key system requirements.

1.1 Functional Requirements

  • Support standard log levels: DEBUG, INFO, WARN, ERROR, and FATAL.
  • Filter log messages based on a configurable minimum log level.
  • Support multiple output destinations (appenders), including console and file.
  • Allow a single log message to be sent to multiple appenders simultaneously.
  • Support asynchronous logging to prevent blocking the main application thread.
  • Allow client applications to configure the logger by specifying log level, formatters, and appenders.

1.2 Non-Functional Requirements

  • Thread Safety: Logging must be safe in concurrent environments to prevent interleaved or lost messages.
  • Performance: Logging should have minimal overhead on application performance.
  • Extensibility: The design should support plugging in custom formatters, filters, and appenders with minimal code changes.
  • Maintainability: The codebase should follow clean, object-oriented design with clear separation of concerns (e.g., logger, formatter, appender).
  • Ease of Use: The client-facing API should be simple and intuitive for developers to use (e.g., logger.info("User logged in");).

2. Identifying Core Entities

Core entities are the fundamental building blocks of our system. We identify them by analyzing the functional requirements and highlighting the key nouns and responsibilities that naturally map to object-oriented abstractions such as classes, enums, or interfaces.

Let’s walk through the functional requirements and extract the relevant entities:

1. The framework should support standard log levels like DEBUG, INFO, WARN, ERROR, and FATAL.

This indicates the need for a LogLevel enum to represent the severity of each log message. It will be used to filter messages based on the configured minimum log level.

Additionally, we need a LogMessage class to encapsulate details of a single log entry such as the message content, timestamp, log level, thread name, and optional metadata.

The Logger class will serve as the primary interface exposed to client applications for logging messages.

2. The system must route messages to one or more output destinations.

This indicates the need for a LogAppender abstraction—an interface for defining output targets.

Concrete implementations might include:

  • ConsoleAppender: Writes logs to the standard output stream
  • FileAppender: Writes logs to a specified file

3. Log entries should support multiple pluggable formats.

This implies the need for a LogFormatter abstraction to define how log messages are serialized before being passed to an appender.

Examples of formatter implementations include:

  • SimpleFormatter: Plain text logs with timestamp and level
  • JSONFormatter: Structured logs for machine parsing or log aggregation systems

4. Logging should be asynchronous to prevent blocking the main application thread.

This requirement introduces the need for an AsyncLogProcessor or LogDispatcher component. It can use a background thread or queue to buffer log messages and dispatch them to appenders in a non-blocking manner.

5. The framework should allow client applications to configure logger behavior.

This suggests a central LogManager entity responsible for creating and managing Logger instances.

These entities define the key abstractions of a logging framework and provide a solid foundation for building a modular, extensible, and performant logging solution.

3. Designing Classes and Relationships

This section outlines the classes, interfaces, and their interactions, which form the architectural backbone of the logging framework.

3.1 Class Definitions

The framework is composed of several types of classes, each with a distinct responsibility.

Enums

LogLevel

A simple enumeration that defines the severity levels for log messages (DEBUG, INFO, WARN, ERROR, FATAL).

LogLevel

It includes a helper method, isGreaterOrEqual(), to easily compare severities and decide if a message should be logged.

Data Classes

LogMessage

An immutable data class (or DTO) that encapsulates all information about a single logging event.

LogMessage

It holds the timestamp, log level, logger name, thread name, and the actual message. This object is passed through the logging pipeline.

Core Classes

Logger

The primary class that application developers interact with.

Logger

It provides methods like info(), warn(), etc., to trigger logging events. It manages its own log level and appenders, and it maintains a link to a parent Logger to form a hierarchy.

AsyncLogProcessor

A background processor responsible for taking LogMessage objects and dispatching them to the appropriate appenders.

AsyncLogProcessor

It uses a single-threaded executor to ensure logs are processed asynchronously without blocking the main application threads and are written in the correct order.

LogManager

A Singleton class that acts as the central point of configuration and management for the entire framework.

LogManager

It is responsible for creating and managing the Logger hierarchy, holding the shared AsyncLogProcessor, and orchestrating a graceful shutdown.

3.2 Class Relationships

The classes interact through a combination of composition, association, and implementation, creating a flexible and extensible system.

Composition

This "has-a" relationship implies ownership, where one object's lifecycle is managed by another.

  • LogManager has a AsyncLogProcessor and a map of all Logger instances. It creates and manages them.
  • Logger has a list of LogAppenders.
  • LogAppender (e.g., ConsoleAppender, FileAppender) has a LogFormatter. The formatter is an integral part of how the appender functions.

Association

This is a weaker "has-a" relationship where objects are related but have independent lifecycles.

  • A Logger has a reference to its parent Logger, forming the core of the logger hierarchy. This is a self-referential association.

Implementation

This relationship exists where a concrete class provides a specific implementation for an interface.

  • SimpleTextFormatter implements the LogFormatter interface.
  • ConsoleAppender and FileAppender implement the LogAppender interface.

Dependency

This relationship exists when one class uses another class.

  • The Logger class depends on LogManager to get the global processor and creates LogMessage objects.
  • The AsyncLogProcessor depends on LogAppender and LogMessage to perform its work.

3.3 Key Design Patterns

Several design patterns are employed to ensure the framework is robust, flexible, and efficient.

Strategy Pattern

This pattern is fundamental to the framework's flexibility.

Formatting Strategy

LogFormatter

The LogFormatter interface allows the algorithm for formatting a log message to be selected at runtime. We can easily add new formatters (e.g., JsonFormatter, XmlFormatter) and assign them to appenders without changing any other code.

Appending Strategy

LogAppender

The LogAppender interface allows the destination for log messages to be selected at runtime. A logger can be configured with multiple appenders to send logs to the console, a file, and a network endpoint simultaneously.

Factory Pattern

The LogManager.getLogger(name) method acts as a factory. It abstracts the creation of Logger instances, intelligently handling the construction of the logger hierarchy by parsing names and linking parents automatically.

Producer-ConsumerPattern

The asynchronous logging mechanism is a classic example of this pattern. Application threads act as Producers, quickly creating LogMessage objects and submitting them. The dedicated AsyncLogProcessor thread acts as the sole Consumer, processing these messages from a queue, thereby decoupling the logging overhead from the application's performance.

Chain of Responsibility Pattern

This pattern is evident in the Logger hierarchy.

  • Level Resolution: To determine if a message should be logged, a Logger checks its own level. If none is set, it delegates the request up to its parent, continuing up the chain to the root.
  • Appender Additivity: A log event is handled by the appenders of the current logger. If additivity is enabled (the default), the event is passed up to the parent's appenders, forming a chain of processing.

Facade Pattern

The Logger class itself serves as a simple facade. It provides a straightforward API (logger.info(...)) that hides the underlying complexity of message creation, level checks, asynchronous processing, formatting, and dispatching to appenders.

Singleton

The LogManager is a singleton, providing a single, globally accessible point for managing loggers and the framework's lifecycle. This prevents conflicting configurations and ensures resources like the thread pool are shared.

3.4 Full Class Diagram

Logging Framework Class Diagram

4. Implementation

4.1 LogLevel Enum

Defines severity levels for log messages.

1class LogLevel(Enum):
2    DEBUG = 1
3    INFO = 2
4    WARN = 3
5    ERROR = 4
6    FATAL = 5
7
8    def is_greater_or_equal(self, other: 'LogLevel') -> bool:
9        return self.value >= other.value

The isGreaterOrEqual() method helps determine whether a log message should be recorded based on the current logger’s configured level.

4.2 LogMessage Class

Encapsulates all details required for a log entry.

1class LogMessage:
2    def __init__(self, level: LogLevel, logger_name: str, message: str):
3        self.timestamp = datetime.now()
4        self.level = level
5        self.logger_name = logger_name
6        self.message = message
7        self.thread_name = threading.current_thread().name
8
9    def get_timestamp(self) -> datetime:
10        return self.timestamp
11
12    def get_level(self) -> LogLevel:
13        return self.level
14
15    def get_logger_name(self) -> str:
16        return self.logger_name
17
18    def get_thread_name(self) -> str:
19        return self.thread_name
20
21    def get_message(self) -> str:
22        return self.message

4.3 LogFormatter and Implementations (Strategy Pattern)

To make the framework flexible, we use the Strategy Pattern to define interchangeable components for formatting logs and sending them to different destinations.

1class LogFormatter(ABC):
2    @abstractmethod
3    def format(self, log_message: LogMessage) -> str:
4        pass
5
6
7class SimpleTextFormatter(LogFormatter):
8    def format(self, log_message: LogMessage) -> str:
9        timestamp_str = log_message.get_timestamp().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
10        return f"{timestamp_str} [{log_message.get_thread_name()}] {log_message.get_level().name} - {log_message.get_logger_name()}: {log_message.get_message()}\n"

The SimpleTextFormatter formats logs into a human-readable format with timestamp, thread, level, logger, and message.

This design decouples the LogAppender (which handles output) from the specific format of the log message. We can easily create new formatters (e.g., JsonFormatter, XmlFormatter) and plug them into any appender without changing the appender's code.

4.4 LogAppender and Implementations

Strategy interface for log destinations.

1class LogAppender(ABC):
2    @abstractmethod
3    def append(self, log_message: LogMessage):
4        pass
5
6    @abstractmethod
7    def close(self):
8        pass
9
10    @abstractmethod
11    def get_formatter(self) -> LogFormatter:
12        pass
13
14    @abstractmethod
15    def set_formatter(self, formatter: LogFormatter):
16        pass
17
18
19class ConsoleAppender(LogAppender):
20    def __init__(self):
21        self.formatter = SimpleTextFormatter()
22
23    def append(self, log_message: LogMessage):
24        print(self.formatter.format(log_message), end='')
25
26    def close(self):
27        pass
28
29    def set_formatter(self, formatter: LogFormatter):
30        self.formatter = formatter
31
32    def get_formatter(self) -> LogFormatter:
33        return self.formatter
34
35
36class FileAppender(LogAppender):
37    def __init__(self, file_path: str):
38        self.formatter = SimpleTextFormatter()
39        self._lock = threading.Lock()
40        try:
41            self.writer = open(file_path, 'a')
42        except Exception as e:
43            print(f"Failed to create writer for file logs, exception: {e}")
44            self.writer = None
45
46    def append(self, log_message: LogMessage):
47        with self._lock:
48            if self.writer:
49                try:
50                    self.writer.write(self.formatter.format(log_message) + "\n")
51                    self.writer.flush()
52                except Exception as e:
53                    print(f"Failed to write logs to file, exception: {e}")
54
55    def close(self):
56        if self.writer:
57            try:
58                self.writer.close()
59            except Exception as e:
60                print(f"Failed to close logs file, exception: {e}")
61
62    def set_formatter(self, formatter: LogFormatter):
63        self.formatter = formatter
64
65    def get_formatter(self) -> LogFormatter:
66        return self.formatter

This design allows us to direct logs to various outputs (console, file, network socket, database) by creating new implementations of LogAppender. A logger can be configured with multiple appenders to send the same log message to several destinations simultaneously.

FileAppender writes logs to a file, handling synchronization and fallback gracefully. Useful for persistent logging in production systems.

4.5 AsyncLogProcessor

To minimize the performance impact on the main application threads, log processing is handled asynchronously.

1class AsyncLogProcessor:
2    def __init__(self):
3        self.executor = ThreadPoolExecutor(max_workers=1, thread_name_prefix="AsyncLogProcessor")
4        self.shutdown_flag = False
5
6    def process(self, log_message: LogMessage, appenders: List[LogAppender]):
7        if self.shutdown_flag:
8            print("Logger is shut down. Cannot process log message.", file=sys.stderr)
9            return
10
11        def process_task():
12            for appender in appenders:
13                appender.append(log_message)
14
15        self.executor.submit(process_task)
16
17    def stop(self):
18        self.shutdown_flag = True
19        self.executor.shutdown(wait=True, timeout=2)
20        if not self.executor._shutdown:
21            print("Logger executor did not terminate in the specified time.", file=sys.stderr)

This class uses a dedicated background thread to format and append log messages, freeing the application thread to continue its work immediately after a log call.

Producer-Consumer Pattern

The application threads act as "producers," submitting LogTask objects to a BlockingQueue. The AsyncLogProcessor runs a single "consumer" thread that takes tasks from the queue and processes them.

Sequential Processing

Using a SingleThreadExecutor ensures that logs are written in the order they were submitted, which is a critical requirement for a logging framework.

4.6 LogManager

Acts as the centralized manager for loggers.

1class LogManager:
2    _instance = None
3    _lock = threading.Lock()
4
5    def __init__(self):
6        if LogManager._instance is not None:
7            raise Exception("This class is a singleton!")
8        self.loggers: Dict[str, 'Logger'] = {}
9        self.root_logger = Logger("root", None)
10        self.loggers["root"] = self.root_logger
11        self.processor = AsyncLogProcessor()
12
13    @staticmethod
14    def get_instance():
15        if LogManager._instance is None:
16            with LogManager._lock:
17                if LogManager._instance is None:
18                    LogManager._instance = LogManager()
19        return LogManager._instance
20
21    def get_logger(self, name: str) -> 'Logger':
22        if name not in self.loggers:
23            self.loggers[name] = self._create_logger(name)
24        return self.loggers[name]
25
26    def _create_logger(self, name: str) -> 'Logger':
27        if name == "root":
28            return self.root_logger
29        
30        last_dot = name.rfind('.')
31        parent_name = "root" if last_dot == -1 else name[:last_dot]
32        parent = self.get_logger(parent_name)
33        return Logger(name, parent)
34
35    def get_root_logger(self) -> 'Logger':
36        return self.root_logger
37
38    def get_processor(self) -> AsyncLogProcessor:
39        return self.processor
40
41    def shutdown(self):
42        # Stop the processor first to ensure all logs are written
43        self.processor.stop()
44
45        # Then, close all appenders
46        all_appenders = set()
47        for logger in self.loggers.values():
48            for appender in logger.get_appenders():
49                all_appenders.add(appender)
50        
51        for appender in all_appenders:
52            appender.close()
53        
54        print("Logging framework shut down gracefully.")

Handles:

  • Logger creation and hierarchy
  • Shared async log processor
  • Graceful shutdown of appenders
  • Singleton Pattern: Ensures a single, globally accessible point for managing loggers and the framework's lifecycle.
  • Logger Factory: The getLogger(name) method is the sole entry point for obtaining a Logger. It cleverly parses the name (e.g., com.example.service) to find its parent (com.example) and recursively calls getLogger to construct the entire hierarchy on demand.

4.7 Logger Class

The main interface for developers.

1class Logger:
2    def __init__(self, name: str, parent: Optional['Logger']):
3        self.name = name
4        self.level: Optional[LogLevel] = None
5        self.parent = parent
6        self.appenders: List[LogAppender] = []
7        self.additivity = True
8
9    def add_appender(self, appender: LogAppender):
10        self.appenders.append(appender)
11
12    def get_appenders(self) -> List[LogAppender]:
13        return self.appenders
14
15    def set_level(self, min_level: LogLevel):
16        self.level = min_level
17
18    def set_additivity(self, additivity: bool):
19        self.additivity = additivity
20
21    def get_effective_level(self) -> LogLevel:
22        logger = self
23        while logger is not None:
24            current_level = logger.level
25            if current_level is not None:
26                return current_level
27            logger = logger.parent
28        return LogLevel.DEBUG  # Default root level
29
30    def log(self, message_level: LogLevel, message: str):
31        if message_level.is_greater_or_equal(self.get_effective_level()):
32            log_message = LogMessage(message_level, self.name, message)
33            self._call_appenders(log_message)
34
35    def _call_appenders(self, log_message: LogMessage):
36        if self.appenders:
37            LogManager.get_instance().get_processor().process(log_message, self.appenders)
38        
39        if self.additivity and self.parent is not None:
40            self.parent._call_appenders(log_message)
41
42    def debug(self, message: str):
43        self.log(LogLevel.DEBUG, message)
44
45    def info(self, message: str):
46        self.log(LogLevel.INFO, message)
47
48    def warn(self, message: str):
49        self.log(LogLevel.WARN, message)
50
51    def error(self, message: str):
52        self.log(LogLevel.ERROR, message)
53
54    def fatal(self, message: str):
55        self.log(LogLevel.FATAL, message)

Each logger:

  • Can define its own level and appenders
  • Inherits behavior from its parent unless overridden
  • Delegates log message processing to LogManager

The additivity flag controls whether messages should propagate up the logger hierarchy.

4.8 LoggingFrameworkDemo  (Driver Code)

The LoggingFrameworkDemo class demonstrates how a client would configure and use the framework.

1class LoggingFrameworkDemo:
2    @staticmethod
3    def main():
4        # --- 1. Initial Configuration ---
5        log_manager = LogManager.get_instance()
6        root_logger = log_manager.get_root_logger()
7        root_logger.set_level(LogLevel.INFO)  # Set global minimum level to INFO
8
9        # Add a console appender to the root logger
10        root_logger.add_appender(ConsoleAppender())
11
12        print("--- Initial Logging Demo ---")
13        main_logger = log_manager.get_logger("com.example.Main")
14        main_logger.info("Application starting up.")
15        main_logger.debug("This is a debug message, it should NOT appear.")  # Below root level
16        main_logger.warn("This is a warning message.")
17
18        # --- 2. Hierarchy and Additivity Demo ---
19        print("\n--- Logger Hierarchy Demo ---")
20        db_logger = log_manager.get_logger("com.example.db")
21        # db_logger inherits level and appenders from root
22        db_logger.info("Database connection pool initializing.")
23
24        # Let's create a more specific logger and override its level
25        service_logger = log_manager.get_logger("com.example.service.UserService")
26        service_logger.set_level(LogLevel.DEBUG)  # More verbose logging for this specific service
27        service_logger.info("User service starting.")
28        service_logger.debug("This debug message SHOULD now appear for the service logger.")
29
30        # --- 3. Dynamic Configuration Change ---
31        print("\n--- Dynamic Configuration Demo ---")
32        print("Changing root log level to DEBUG...")
33        root_logger.set_level(LogLevel.DEBUG)
34        main_logger.debug("This debug message should now be visible.")
35
36        try:
37            time.sleep(0.5)
38            log_manager.shutdown()
39        except Exception as e:
40            print("Caught exception")
41
42if __name__ == "__main__":
43    LoggingFrameworkDemo.main()

The driver code showcases the framework's key features:

  • Central Configuration: Setting a level on the root logger applies globally.
  • Hierarchical Usage: Getting loggers by name (com.example...) automatically links them in the hierarchy.
  • Granular Control: Overriding the log level for a specific logger (dbLogger) without affecting others.
  • Graceful Shutdown: Ensuring all buffered logs are written before the application exits.

5. Run and Test

Languages
Java
C#
Python
C++
Files11
entities
enum
strategies
async_log_processor.py
log_manager.py
logger.py
logging_framework_demo.py
main
logging_framework_demo.py
Output

6. Quiz

Design Logging Framework - Quiz

1 / 21
Multiple Choice

Which component in a logging framework is responsible for sending log messages to a specific destination such as a file or console?

How helpful was this article?

Comments


0/2000

No comments yet. Be the first to comment!

Copilot extension content script